Multi-point predictive foveation for bandwidth reduction of moving images

ABSTRACT

A scene of an image sequence is transmitted that has been compressed using a number of foveation zones, each foveation zone being weighted based on a probability of a viewer looking at a corresponding portion of the first scene.

RELATED APPLICATIONS

The present patent document is a continuation of U.S. patent applicationSer. No. 11/313,335, filed Dec. 21, 2005, now U.S. Pat. No. 7,251,373which is a continuation of U.S. patent application Ser. No. 10/123,061,filed Apr. 15, 2002, now U.S. Pat. No. 7,010,169 the entirety of each ofwhich is hereby incorporated by reference.

BACKGROUND

1. Field of the Invention

The present invention relates to image compression methods which usefoveation.

2. Description of the Related Art

Foveation is a compression method in which an image is compressed sothat it matches an ability of a human visual system to detect detail ina peripheral visual field. Methods of image compression using foveationare disclosed in U.S. Pat. No. 6,252,989 to Geisler and Kortum.Foveation is a powerful method of image compression because of highcompression rates and perceptually lossless images which result.Drawbacks of the method include requiring real-time knowledge of where auser is looking, and not easily supporting multi-viewer environments.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is pointed out with particularity in the appendedclaims. However, other features are described in the following detaileddescription in conjunction with the accompanying drawings in which:

FIG. 1 is a flow chart of an embodiment of a method of multi-pointpredictive foveation;

FIG. 2 is a block diagram of an embodiment of a system for multi-pointpredictive foveation;

FIG. 3 shows an example of an unfoveated image;

FIG. 4 shows the image with a simulated single point foveation; and

FIG. 5 shows the image with a simulated multipoint foveation.

DETAILED DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention employ multiple zones of foveationwhich facilitates use in a multi-viewer environment without real-timeknowledge of each viewer's gaze. Each foveation zone is weighted basedon a probability of a viewer looking at a specific location in a scene.Center points of the foveation zones may be determined eitheralgorithmically or empirically. Implementation of the multi-pointfoveation compression scheme allows movies of significantly higherquality to be delivered over a telecommunication network, such as aDigital Subscriber Line (DSL) network, without a corresponding increasein required bandwidth.

Embodiments of the present invention are described with reference toFIG. 1, which is a flow chart of an embodiment of a method ofmulti-point predictive foveation, and FIG. 2, which is a block diagramof an embodiment of a system for multi-point predictive foveation.Consider an image sequence 10 depicted in FIG. 2. Examples of the imagesequence 10 include, but are not limited to, all or part of a televisionprogram, a movie, a live video event, an output of a Web camera, andother video events which provide moving images. The events captured inthe image sequence 10 may be either live or recorded.

The image sequence 10 comprises a plurality of scenes. For example, theimage sequence 10 may include scenes 12, 14, 16 and 20. The differentscenes may be defined by different cameras, different perspectives of acamera, different periods in time, different locations, and/or differentobjects captured in the images. As depicted in FIG. 2, the differentscenes 12, 14, 16 and 20 may consist of different numbers of imagesand/or have different lengths of time.

The image sequence 10 is to be compressed either to reduce a bandwidthrequired to transmit a compressed version thereof or to reduce an amountof data required to store the compressed version thereof. For example,the compressed version may be transmitted by a telecommunication network22 to one or more viewers. Examples of the telecommunication network 22include, but are not limited to, a digital subscriber line (DSL)network, a digital cable television transmission network, a satellitetransmission network, and a computer network such as an internet or anintranet. Alternatively, the compressed version may be stored to astorage device 24. Examples of the storage device 24 include, but arenot limited to, an optical storage medium such as a digital video disk(DVD) or a compact disk (CD), a magnetic storage medium such as amagnetic disk, and an electronic storage medium such as a memory card.

Referring to block 26 in FIG. 1, a series of acts is performed for eachscene in the image sequence 10. As indicated by block 30, an act ofdetermining at least one foveation zone is performed for a scene in theimage sequence 10. Each foveation zone may be determined eitherempirically or algorithmically.

To empirically determine the foveation zones, the image sequence 10 isdisplayed to a sample of viewers 32 using a display device 34. One ormore eye tracking devices 36 sense where within each of the differentscenes each of the viewers 32 is viewing. For the purpose of thisdisclosure, the portion of a scene that a viewer is viewing is called a“view location”. The viewers 32 may be either simultaneously watchingthe image sequence 10 or watching the image sequence 10 at differenttimes.

The view locations within each scene are determined for the sample ofviewers 32 using the eye tracking devices 36. A processor 40 isresponsive to view location data from the eye tracking devices 36 todetermine one or more foveation zones for each scene. The processor 40determines the foveation zones based on clusters of the view locations.

To algorithmically determine the foveation zones, the sample of viewers32, the display 34, and the eye tracking devices 36 may be omitted. Incontrast, the foveation zones are determined based on a scene type andobjects in the scene. The processor 40 performs acts to determine thefoveation zones based on computer program code which assists inrecognizing the scene type and objects in the scene. Examples ofdifferent scene types include, but are not limited to, a high actionscene and a conversation scene. Examples of different objects in a sceneinclude, but are not limited to, humans in a scene, and moving objectsin a scene. The processor 40 may assign one foveation zone to each highaction scene, and two foveation zones to a conversational scene betweentwo people, for example.

To illustrate the determination of different foveation zones, considerthe scenes 12, 14, 16 and 20 in the image sequence 10. For purposes ofillustration and example, the scene 12 comprises a sequence of images ofhaving two main foreground objects—a human and a dog in a room; thescene 14 comprises a sequence of images of a conversation between thehuman and another human in the room; the scene 16 comprises a sequenceof images of a fight between the two humans in the room; and the scene20 comprises a sequence of images of the human running out of the roomafter the fight.

Either empirically or algorithmically, it is determined in the aboveexample that: the scene 12 has two foveation zones—one for the human andone for the dog; the scene 14 has two foveation zones—one for eachhuman; the scene 16 has one foveation zone at a central portion of thefighting action; and the scene 20 has one foveation zone which followsthe human as he runs out of the room.

In general, an image sequence will have scenes with different numbers offoveation zones. In other words, some scenes will have either more orfewer foveation zones than other scenes in the image sequence. Also, itis noted that some scenes may have three or more foveation zones.

As indicated by block 42, a corresponding probability of a viewerlooking at a corresponding portion of the scene is estimated for eachfoveation zone. Each probability may be determined either empirically oralgorithmically. Empirically, the probability may be based on theproportion of the viewers 32 whose view locations are within acorresponding foveation zone. Algorithmically, the probability may bedetermined by the processor 40 based on an expected proportion ofviewers whose view locations will be within a corresponding foveationzone. The expected proportion may based on the scene type and objects inthe scene.

Continuing with the above example, either empirically or algorithmicallythe following probabilities are assigned to the various foveation zonesin the image sequence 10. In the scene 12, the foveation zone associatedwith the human has a probability of 0.8 and the foveation zoneassociated with the dog has a probability of 0.1. In the scene 14, thefoveation zone associated with the human has a probability of 0.5 andthe foveation zone associated with the other human is 0.5. In the scene16, the single foveation zone has a probability of about 1. In the scene20, the single foveation zone has a probability of about 1.

As indicated by block 44, the method comprises weighting each foveationzone based on its corresponding probability. The foveation zones areweighted so that more resolution is associated with portions of thescenes having a greater probability of being viewed. In general, thefoveation zones may be definable by either a continuous falloff functionor a discrete function. Examples of specific types of foveationfunctions are disclosed in U.S. Pat. No. 6,252,989 to Geisler andKortum, which is hereby incorporated by reference into the presentdisclosure.

For a continuous falloff function, each foveation zone may have acorresponding half-resolution constant based on the probability of aviewer looking at the corresponding portion of the scene. Thus, for ascene having a first foveation zone and a second foveation zone, wherethe first foveation zone has a greater probability of a viewer lookingat its corresponding portion than the second foveation zone, thehalf-resolution constant of the first foveation zone is greater than thehalf-resolution constant of the second foveation zone. For example, inthe scene 12, the half-resolution constant associated with the human isgreater than the half-resolution constant associated with the dog.

For a discrete falloff function, each foveation zone may have acorresponding fixed resolution based on the probability of a viewerlooking at the corresponding portion of the scene. Thus, for a scenehaving a first foveation zone and a second foveation zone, where thefirst foveation zone has a greater probability of a viewer looking atits corresponding portion than the second foveation zone, the resolutionof the first foveation zone is greater than the resolution of the secondfoveation zone. For example, in the scene 12, the resolution of thehuman may fixed at 2400-by-1100 pixels, the resolution of the dog may befixed at 1200-by-700 pixels, and the resolution of the remainder ofscene may be fixed at 640-by-480.

As indicated by block 46, the method comprises compressing each sceneusing its corresponding at least one foveation zone. The act ofcompressing is performed by a compressor 50. By compressing all of thescenes, the compressor 50 generates a compressed version of the imagesequence 10. The compressed version may be stored to the storage device24 and/or transmitted (block 52) by a transmitter 54 in communicationwith the telecommunication network 22. Examples of the transmitter 54include, but are not limited to, a modem, a computer network interface,a radio frequency transmitter, an optical transmitter, and otherwireless and wireline transmitters.

For purposes of illustration and example, consider the telecommunicationnetwork 22 comprising a DSL network, and the image sequence 10comprising a movie. The compressed version of the movie is communicatedvia the DSL network to a plurality of different viewer's premises,including a viewer's premise 56 and a viewer's premise 60. The premise56 has a receiver 62 which receives the compressed version via the DSLnetwork, an optional decompressor 64 which decompresses the compressedversion, and a display 66 to display the movie to a viewer 70.Similarly, the premise 60 has a receiver 72 which receives thecompressed version via the DSL network, an optional decompressor 74which decompresses the compressed version, and a display 76 to displaythe movie to a viewer 80. In general, each of the receivers 62 and 72may be embodied by a modem, a computer network interface, a radiofrequency receiver, an optical receiver, or other wireless or wirelinereceivers. Each of the optional decompressors 64 and 74 may be embodiedby a general purpose computer, for example. Each of the displays 66 and76 may be embodied by a computer monitor or a television, for example.

Typically, the decompressors 64 and 74 are used in embodiments where theimage sequence is compressed by both foveated compression and anotherform of compression, e.g. an MPEG (Moving Pictures Expert Group)standard such as MPEG4. However, in embodiments where the image sequenceis compressed using substantially only foveated compression, thedecompressors 64 and 74 may be omitted.

It is noted that the viewers' 70 and 80 view locations within the movieare not necessarily monitored by eye tracking devices. Thus, thecompression of the movie is independent of viewing behavior of theviewers 70 and 80. However, the compression of the movie is based on apredicted behavior of the viewers 70 and 80. For example, in the scene12, the viewer 70 may be looking at the human while the viewer 80 islooking at the dog. For a typical movie, the entire movie is compressedbased on the eye behavior of the sample of viewers 32 before the viewers70 and 80 have access to the movie. However, in live event applications,the video may be compressed in real-time based on the eye behavior ofthe sample of viewers 32. Thus, the viewers 70 and 80 would have aslightly-delayed access to the compressed version of the live video.

FIGS. 3 to 5 illustrate how different numbers of foveation zones affectan image. FIG. 3 shows an example of an unfoveated image. FIG. 4 showsthe same image with a simulated single point foveation at the “X”. Theresulting foveation zone about the “X” is more detailed than theremainder of the image. Further, the foveation zone is defined by acontinuous falloff function which matches the ability of the humanvisual system to detect detail in the peripheral visual field. FIG. 5shows the same image with a simulated multi-point foveation. One pointof foveation is at the “X” and another point of foveation at the “A”.The resulting foveation zones about the “X” and “A” are more detailedthan the remainder of the image. In a dynamic environment and at aproper viewing distance, all three of these images would looksubstantially identical to a viewer whose gaze is directed toward the“X”.

It is noted that the processor 40 disclosed herein may be provided by ageneral purpose microprocessor or a custom processor. The functionalityprovided by the processor 40 may be implemented in hardware and/orsoftware. The processor 40 may be responsive to a computer-readablemedium having computer-readable instructions such as computer programcode to direct the acts described with reference to FIG. 1. Typically,the processor 40 is provided by either a general purpose computer systemor an application-specific computer system, which is also programmed toprovide the functionality of the compressor 50.

It will be apparent to those skilled in the art that the disclosedinventions may be modified in numerous ways and may assume manyembodiments other than the preferred forms specifically set out anddescribed herein. For example, the acts described with reference to FIG.1 may be performed in an order which differs from the order shown in theflow chart. Further, some of the acts may be performed in parallel.

Accordingly, it is intended by the appended claims to cover allmodifications which fall within the true spirit and scope of the presentinvention.

1. A method comprising: transmitting an image sequence that iscompressed based on a plurality of foveation zones, wherein a firstfoveation zones of the plurality of foveation zones corresponds to afirst view location and a second foveation zone of the plurality offoveation zones corresponds to a second view location; wherein during aportion of the image sequence, the first and second view locations occurconcurrently.
 2. The method of claim 1, further comprising: compressingthe image sequence based on the plurality of foveation zones.
 3. Themethod of claim 2, further comprising: determining view locations withinthe plurality of foveation zones for a sample of viewers.
 4. The methodof claim 2, further comprising: estimating a probability of a viewerlooking at the first view location and the second view location based atleast in part on a scene type occurring during the portion of the imagesequence where the first and second view locations occur concurrently;and weighting at least the first and second foveation zones based on theprobability.
 5. The method of claim 1 wherein each of the plurality offoveation zones is definable by a continuous falloff function.
 6. Themethod of claim 1 wherein each of the plurality of foveation zones has acorresponding half-resolution constant based on a probability of aviewer looking at the view location corresponding the foveation zone. 7.The method of claim 6 wherein the probability of a viewer looking at thefirst view location corresponding to the first foveation zone is greaterthan the probability of a viewer looking at the second view locationcorresponding to the second foveation zone, and wherein thehalf-resolution constant of the first foveation zone is greater than thehalf-resolution constant of the second foveation zone.
 8. The method ofclaim 1 wherein each of the plurality of foveation zones is definable bya discrete function.
 9. The method of claim 1 wherein each of theplurality of foveation zones has a corresponding fixed resolution basedon a probability of a viewer looking at a view location corresponding tothe foveation zone.
 10. A computer-readable medium havingcomputer-readable instructions to direct a computer to perform an actof: transmitting an image sequence that is compressed based on aplurality of foveation zones, wherein a first foveation zones of theplurality of foveation zones corresponds to a first view location and asecond foveation zone of the plurality of foveation zones corresponds toa second view location; wherein during a portion of the image sequence,the first and second view locations occur concurrently.
 11. Thecomputer-readable medium of claim 10, further comprisingcomputer-readable instructions to direct the computer to perform an actof: compressing the image sequence based on the plurality of foveationzones.
 12. The computer-readable medium of claim 11 wherein thecomputer-readable instructions further direct the computer to performacts of: estimating a probability of a viewer looking at the first viewlocation and the second view location based at least in part on a scenetype occurring during the portion of the image sequence where the firstand second view locations occur concurrently; and weighting at least thefirst and second foveation zones based on the probability.
 13. Anapparatus operable to transmit an image sequence that is compressedbased on a plurality of foveation zones, wherein a first foveation zonesof the plurality of foveation zones corresponds to a first view locationand a second foveation zone of the plurality of foveation zonescorresponds to a second view location; wherein during a portion of theimage sequence, the first and second view locations occur concurrently.14. The apparatus of claim 13, wherein the apparatus is further operableto compress the image sequence based on the plurality of foveationzones.
 15. The apparatus of claim 14, wherein the apparatus comprises:at least one eye tracking device to determine view locations for asample of viewers; and a processor responsive to the at least one eyetracking device to determine the plurality of foveation zones based onthe view locations, to estimate a probability of a viewer looking at theview locations, and to weight each of the foveation zones based on theprobability of the viewer looking at the view locations.